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ABSTRACT 

A sample of 508 educational policymakers (state 
legislators, state directors of vocational education, local 
vocational administrators, and directors of state councils of 
vocational education) was asked to rate the relative importance of 38 
quality indicators for secondary vocational education. Usable data 
were obtained from 207 (67*/.) of the surveys. Of the 10 indicators 
ranked most important, 3 measured student achievement and learning, 3 
measured labor market outcomes, 2 measured student attendance and 
retention, and the remaining 2 measured student educational 
advancement and employer satisfaction. Of the 10 indicators ranked 
most feasible, 3 were measures of student achievement and learning, 3 
were measures of labor market outcomes, 2 were measures of student 
attendance and retention, and 2 were measures of student educational 
advancement and employer satisfaction. Seventeen indicators were 
identified as having both relatively high importance and feasibility 
ratings. High levels of agreement were found among respondent groups 
as to the importance of the individual quality indicators. An 
exploratory factor analysis revealed that the respondents considered 
the following factors most important: employer satisfaction, economic 
and socicl benefits, employment rates, cognitive achievement, and 
individual and institutional educational effort. Sixteen 
recommendations regarding future practical and theoretical research 
were formulated. (Contains 22 references.) (MN) 
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The Ohio State University, Columbus, Ohio 43210-1067 

Ratings of Quality Indicators for Secondary Vocational Education 
Programs by Educational Policy Makers 

Donald D. Peasley and N. L. McCaslin 



Introduction 

An analysis of the recent discourse on American 
public education shows that issues of quaUty and 
accountability received more attention than any other 
issue. A host of national reform reports— A Nation at 
Risk (National Commission on Excellence in Educa- 
tion, 1983), Workforce2000 (Johnston, 1987), America 
2000 (U. S. Department of Education, 1990), Educa- 
tionCounts (U.S.Departmentof Education, 1991) — 
have carried a theme of holding schools accountable 
for student achievement, attainment, and retention of 
requisite knowledge, skills, and attitudes. However, 
while every significant report on educational reform in 
the past 10 years has maintained that school systems 
must be held accountable, considerably less agree- 
ment exists on how this accountability should be mea- 
stired. 

Vocational education has been deeply involved in 
policy debates regarding quaUty and accountability. 
EJvaluating vocational education program quality is not 
new (Taylor, in Darcy, 1979). Recent legislation (PL 
101-392) explicitly required that states develop and 
implement a statewide system of core standards and 
measures for evaluating local secondary programs. 
Hochlander and Rahn (1992) reported that, in 199 1 , all 
50 states p lanned to design systems to measure voca- 
tional education programperformance. McCaslin and 
Headley (1993) found that by early 1993, nearly all 
states had implemented at least part of those systems. 

These systems, designed to measure program 
quality, were based upon a variety of indicators. The 
development of indicators which measure social phe- 
nomena is a complex task which requires empirical 
validation (DcNeufville, 1975). The knowledge base on 



vocational education in the United States is rich in 
conceptual supposition regarding indicators of program 
quality , but poor in terms of empirical evidence that 
supports the efficacy of these indicators. 

Problem Statement 

Copa and Salem (1982) conducted a study which 
identified several potentially vaUd indicatorsfor use in 
evaluating vocational education programs, based on 
importance ratings of Minnesota vocational education 
supervisors. Theyrecommendedthatfurtherresearch 
investigate thevaUdity, feasibility, andconceptual co- 
herence of vocational education indicators. To date, 
there have been no studies of a national scope which 
have attempted to validate the types of indicators used 
in evaluatingvocational education programs. Ifvoca- 
tional education decision makers are going to use 
indicators for program improvement , then a ss^stematic 
effortmustbe undertaken to determine which indica- 
tors are most relevant and useful. Leaders and schol- 
ars in the discipline of vocational education must 
participate in scholarship which advances this effort. 

Purpose of the Research and 
Research Questions 

The purpose of this research was to validate a Lbt 
of indicator statementsfor use in evaluatingsecondary 
vocational education programs. Also, this study was an 
exploratory attempt to establish the initial empirical 
coherence of McCaslin's (1990) fi-ameworkfor evaluat- 
ing secondary vocational education programs As part 
of the investigation, relevantvocationaleducation policy 
makers (Federal legislators, state legislators, state 
directors of vocational education, local vocational ad- 
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ministxators, andexecutive directors of state councils 
of vocational education) were asked to rate a series of 
indicator statements related to secondary vocational 
education on the criteria of importjuice and feasibility . 
The ratings were then used to develop aparsimonious 
list of indicators which had utility in evaluating and 
Eissessing secondary vocational education programs. 

The following research questions were developed 
to guide the study. 

1) Which indicators of secondary vocational educa- 
tion quality rate as important to vocational educa- 
tion policy makers? Importance was defined as 
"a quality, character, or standing such as to 
entitle to attention or consideration" of an object 
(Random House, 1979, p. 668). Importance was 
operationally defined in this study as the nu- 
merical ranking (1 being the lowest, 5 the high- 
est) assigned to a quality indicator statement 
assigned by a vocational education policy maker. 

2) How feasible are indicators of secondary voca- 
tional education quality, as rated by vocational 
education p olicy makers? Feasibility referred to 
the practicality or "do ability " of a task or object 
(Random House, 1979, p. 483). Feasibility was 
operationally defined in this study as the nu- 
merical ranking (1 being the lowest, 5 the high- 
est) assigned to a quality indicator statement 
assigned by a vocational education policy maker. 

3) What is the relationship between the feasibility of 
secondary vocational education quality indicators 
and their importance, as rated by vocational educa- 
tion policy makers? 

4) Are there differences between the different groups 
of vocational education policy makers in terms of 
the importance and feasibility ratings of secondary 
vocational education quality indicators? 

5) Whatarethefactorsthatunderlietheimportanceof 
secondary vocational education quality indicators , 
as rated by vocational education policy makers? 

6) Are the factors that underlie the importance ratings 
of secondary vocational education quality indica- 
tors congruent with McCaslin's ( 1990) evaluation 
framework for vocational education programs? 

Theoretical/Conceptual 
Framework 

Research and development of indicators in the 
field of education increased dramatically during the 
1980s (Bryk&Hermanson, 1993). In 1984, the U.S. 
DepartmentofEducation published thefirst Wall Chart. 
which was a compilation of educational statistics and 
test scores for each of the 50 states. Debate over the 
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efficacy of these statistics as adequate indicators led 
to a surge in indicator development (Burstein, Oakes, 
& Guiton, 1992). The literature on the use of 
educational indicators has particularly stressed the 
notion of systems of indicators. Raizen and Jones 
(1985) in one of the first research based efforts to 
develop educational indicators, recommended that 
: \irther research should study specific indicator sys- 
tems. These systems should, in theory, better 
describe the components of the educational enter- 
prise in greater depth than individual measures. 

While much of the literature about educational 
indicators supports the development of systems of 
indicators, vocational education has primarily used 
individual indicators (White. 1990). Copa and Scholl 
(1983) stressed the need for the construction of sys- 
tems of indicators to evaluate vocational education 
programs. Evaluations of vocational education must 
rely on systems of evaluativ e information which reflect 
the complex interrelationships of inputs, processes, 
and outcomes (Asche, 1990). A necessary first step in 
conducting research on educational indicators is to 
identify a conceptual model of the educational system 
(Burstein, Oakes, & Guiton, 1992). 

Vocational education literature is rich on evalua- 
tive models for vocational education. One common 
theme among all of these models is that they seek to 
describe educational inputs, processes, andoutcomes. 
Public policy has shifted the emphasis on vocational 
education evaluation to program outcomes (White, 
1990). 

McCaslin (1990) proposed a conceptual firame- 
work for evaluating vocational education programs 
(Figure 1). The basic premise of the framework was 
that those who wish to asuessprc^am outcomes must 
consider a more comprehensive approach. McCaslin 
posited that there were three major types of evaluative 
information. Information about educationalneeds for 
vocational education included those needs expressed 
by the "clients" of vocational education — students, 
employers, and society. The second type of informa- 
tion represented the processes of vocational educa- 
tion. The final component of the framework was the 
outcomes — economic, educational, andpsychosocial — 
of vocational education. Eachcomponentoftheframe- 
work was thought tobe related in a dynamic fashion to 
the other part of the system. Each component was 
comprised of several distinct dimensions. Indicators 
provide measures of these dimensions for each com- 
ponent in the system. The process and outcome com- 
ponents of this fr2miework were used as the basic 
conceptual foundation of this research. 
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Figure 1. Framework for evaluating vocational education . (McCaslin, 1990, p. 10) 
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Methodology 

There were five distinct populations in this 
research. The first was state legislators involved in 
vocational education policy making. The accessible 
population were those legislators who chaired edu- 
cation committees in their states. A census (N=100) 
of these legislators was taken for this study. The 
second population for the study was Federal level 
legislators most involved with vocational education 
policy. A census (N=84) was taken of members of the 
four committees most relevant to vocational educa- 
tion policy: 1) the U.S. House Appropriations Com- 
mittee-Subcommittee on Labor, Health & Human 
Services; 2) the U.S. House Committee on Education 
and Labor (N=41); 3) theU.S. Senate Appropriations 
Committee-Subcommittee on Labor, Health & Hu- 
man Services and Education (N=13); and 4) the U.S. 
Senate Committee on Labor and Human Resources 
(N=17). The thirdpopulation for this study was state 
directors of vocational education. A census (N=55) of 
these individuals was taken. The fourth population 
for this research was executive directors of state 
councils of vocational education. Acensus (N=53) of 
these individuals was drawn. The fifth population 
for this research was local administrators of voca- 
tional education. The frame for this study were 
members of the National Council of Local Adminis- 
trators (NCLA), a national organization of voca- 
tional administrators (N=l,387). A random sample 
(N= 100) was drawn from this frame.A random sample 
was used for this group because of the large size of 
the population and because of a desire to construct 
roughly equivalent groups of policy-makers. 

The instrumentation for this research consisted of 
a written questionnaire containing 38 indicator state- 
ments related to vocational education processes and 
outcomes (see Table 1). These indicator statements 
were developed from a review of the literature re- 
garding vocational education evaluation. Respon- 
dents rated each statement from one Gowest) to five 
(highest) on two dimensions-importance and feasi- 
bility. The scaling technique utilized was a modifi- 
cation of a "double-barreled" pre-needs assessment 
survey design (Witkin, 1984). 

Content validity of the instrument was estab- 
lished through a panel of experts, comprised of 
vocational education teacher educators, and state- 
level vocational education administrators. The in- 
strument was also reviewed for format and scaling 
considerations, face validity, and respondent bur- 
den. Instructions and item content were revised 
accordingly after reviewing the comments of the 
panel of experts. Item reliability was estimated 
through an extensive pilot testing procedure. 
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Vocational teacher educators, state level voca- 
tional education officials, and graduate students en- 
rolled in a graduate-level vocational education plan- 
ning course (n=28) comprised thepilot testing sample. 
Test-retest percentages of for the items on the instru- 
ment ranged from 71% to 100%, and averaged 85%. 

Data were collected by mail survey. Cover letters 
and questionnaires were sent out the second week in 
June, 1993. After the second mailing, only seven out 
of 84 (8%) of the Federal legislators had responded. 
Due to the poor response rate the decision was made 
at that time to omitFeder?! legislators from thestudy. 
Usable data was obtained from 207 (67%) of the 
remaining 308 individuals. No attempt was made to 
make probabilistic inferences beyond these respon- 
dents. Data collected were directly applicable only to 
those 207 individuals who responded to the study. 
Consequently, conclusions and recommendations 
made in this study are applicable only to those same 
individuals. By the same token, the respondents to 
the survey do represent important members of voca- 
tional education policy makers at state and local 
levels. Logically, their responses to the survey 
should at the least be of interest to all involved with 
vocational education. 

Summary of Findings 
Research Question One 

The top lOmostimportantindicators rated by the 
overall respondents are listed in Table 2. Three of 
these indicators were labor market outcomes Gabor 
forceparticipation, occupation relatedplacement, and 
overall employment rates). Another three statements 
directly reflected indicators of employer satisfaction 
(with vocational education program graduates). Two 
indicators weremeasures of student achievement, and 
the final two indicators were measures of student 
educational advancement. Executive directors of state 
councils of vocational education and state legislators 
rated two and three indicators, respectively, among 
their 10 most important which werenot rated amongthe 
top 10 most important overall. In general, the data 
collected for this research question showed thatmea- 
sures of employer satisfaction . student achievement, 
labor marketoutcomes, and theeducational advance- 
ment app ear to be imp ortant to aU policy makers who 
responded to the survey. 

Research Question Two 

The 10 most feasible indicators rated by the 
overall group of respondents are listed in Table 3. 
Three of these indicators were measures of student 



Table 1 

List of Thirty-Eight Indicator Statements Used in the Survey 



Item Indicator Statement 



1. Student achievement measured by a gaia on a standardized test (pretest-posttest). 

2. Student retention measured by hours of program enrollment. 

3. A description of the counseling services available to the student. 

4. A description of the recruitment activities conducted by the program. 

5. Evidence of teacher performance measured by student opiaion survey. 

G. A description of how program evaluation results feed iato program planniag. 

7. The rate of student advancement to a higher level of skiU or competency ia the program. 

8. Student achievement measured by gaia on a competency (skills) based test (pretest-posttest). 

9. Student lejimiag measured by teacher reports of gaias and improvements. 

10. Student retention measured by the percent of students who complete the program. 

11. A comparison of program student characteristics with those of the target student profile. 

12. Student pairticipation ia the labor force after completiag the program. 

13. Student achievement as measured by assessment of student work portfolios. 

14. Employer satisfaction of program graduates training measured by opiaion survey. 

15. The economic return on iavestment of program graduates as measured by the ratio of gradu- 
ate eamiags to profpram costs. 

16. Student perceptions of the quality of their vocational preparation. 

17. The rate of student employment ia jobs directly related to their trainiag. 

18. Student job success as measured by the average weekly wage of program graduates. 

19. The emplojnment rate of program graduates. 

20. The rate of program graduates who receive government assistance (ADC, general relief, or 
food stamps). 

21. Student achievement as measured by gaia of students' basic skills on a standard proficiency 
test (pretest-posttest). 

22. The career Eispirations of program graduates measured by opiaion survey. 

23. Student citizenship as measured by the rate of graduate voter registration 

24. Employer satisfaction of program graduates work habits measured by opiaion survey. 

25. Employer satisfaction of program graduates work competency measured by opiaion survey. 

26. Evidence of teacher performance measured by supervisor ratiags. 

?7. Critical thiaking ability of program graduates as measured by a standardized critical thiaking 
test. 

28. Problem solving ability of program graduates measured by employer appraisal and teacher 
observations. 

29. Educational aspirations of program graduates measured by opiaion survey. 

30. Student attendance measured by average days per school year missed by students. 

31. The percentage of male and female program graduates employed across all types of occupa- 
tions. 

32. The earnings of program graduates employed in jobs directly related to their training. 

33. The percentage of program graduates who are self-employed as entrepreneurs. 

34. The percentage of program graduates who are "ontiauiag their education in programs related 
to their trainiag. 

35. Program graduates' knowledge of the world of work as measured by teacher based tests and 
observation. 

36. Program graduates' self-esteem as measured by a student self-appraisal on an opiaion survey. 

37. Program graduates' leadership development as measured by participation rates in vocational 
student organizations. 

38. Program graduates' work ethic as measured by survey of employers and teacher observation. 
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Table 2 

Ten Most Important Indicators, as Rated by the Overall Responding Sample (n=207) 
Indicator Statement Mean Rating Std. Deviation 



Student participation in the labor force after completing the 
program. 

Employer satisfaction of program graduates traiauig 

measured by opiaion survey. 
Student achievement measured by gain on a competency 

(skills) based test (pretest-posttest). 
The employment rate of program graduates. 
Employer satisfaction of program graduates work competency 

measured by opinion survey. 
The percentage of program graduates who are continuing their 

education in programs related to their training. 
The rate of student advancement to a higher level of skill or 

competency in the program. 
Employer satisfaction of program graduates work habits 

measured by opinion survey. 
The rate of student emplosmaent in jobs directly related to 

thfcir training. 

Student achievement as measured by assessment of student 
work portfolios. 



4.5 


.8 


4.4 


.8 


4.4 


.9 


4.3 


.9 


4.2 


.8 


4.1 


.9 


4.1 


1.0 


4.1 


1.0 


4.1 


1.0 


4.1 


.9 



Note: Ratings based upon a five point rating scale where l=lowest and 5=highest. 
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achievement and learning, three were measures of 
labor market outcomes, two were measures of stu- 
dent attendance and retention, and the remaining 



two were measures of student educationeil advance- 
ment and employer satisfaction. Overall, measures 
of student attendance, retention , achievement, leam- 



Table3 

Ten Most Feasible Indicators, as Rated by the Overall Responding Sample (n=207) 



Indicator Statement 



Mean Rating Std. Deviation 



Student retention measured by the percent of students who 

complete the program. 
Student attendance measured by average days per school year 

missed by students. 
The employment rate of program graduates. 
Student participation in the labor force after completing the 

program. 

Student achievement measured by gain on a competency (skills) 

based test (pretest-posttest). 4.0 1.0 
The percentage of program graduates who are continuing their 

education in programs related to their training. 
The rate of student employment in jobs directly related to their 

training. 

Employer satisfacti'^n of program graduates training measured by 
opinion survey. 

Student achievement measured by a gain on a standardized test 

(pretest-posttest) . 
Student learning measured by teacher reports of gains and 

improvements. 



4.4 

4.1 
4.0 

4.1 



4.0 
4.0 
3.9 
3.9 
3.8 



Note: Ratings based upon a five point rating scale where l=lowest and 5=highest. 



LI 

1.1 
.9 

1.0 



.9 
1.0 
1.0 
L2 
1.0 
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Table 4 








Spearman Rank Order Correlations of Respondent Groups Importance Rankings of 

Indicator Statements 


Respondent Group 


I. 


2. 


3. 


4. 5. 


1. Overall Group Rank 

2. Executive Directors SCOVEs 

3. State Legislators 

4. State Directors of Voc. Educ. 

5. Local Admin. 


.96 
.91 
.97 
.96 


.81 
.92 
.90 


.84 
.89 


.89 — 



ing, labor market outcomes, and employer satisfaction 
were rated as being the most feasible indicators for 
evaluating secondary vocational education programs. 

Research Question Three 

Seventeen indicators were identified as having 
both relatively high importanceandfeasrbility ratings, 
that is, each of these 17 indicatorshadratinpthatwere 
above both the overall mean rating for importance (3. 6) 
emd feasibility (3.5).Three measures of labor out- 
comes, three measures of employer satisfaction, and 
three measures of student achievement were rated 
as having the greatest potential for use in evaluating 
vocational education. Additionally, two measures of 
student educational advancement and individual 
measures of student learning, retention, satisfac- 
tion , work ethic, and school planning were identified 
by this analysis. 

Research Question Four 

There were high levels of agreement between the 
respondent groups in termsof their importance ranking 
of ■"ocational education evaluation indicator state- 
ments, as evidencijd by the Spearman rank order 
correlations listed in Table 4. Generzdly, there was 
a high level of agreement between the respondent 
groups in terms of their feasibility rankings of the 
vocational education evaluation indicator statements 



(Table 5). Another index of the level of agreement 
between the respondent groups is Kendall coeffi- 
cient of concordance W. This measure describes the 
overall agreement between each set of respondent 
group ranks. For both the importance andfeasibility 
rankings, Kendall W was substantially positive at 
.91 and .84, respectively. 

Research Question Five 

An exploratory factor analysis (usingthecommon 
factor model; with oblique rotation) was performed in 
order to determine underlsong factors with respon- 
dents' importance ratings of the indicator state- 
ments. An initial factor analysis was performed in 
order to determine the optimum number of factors to 
select for extraction. This determination was based 
upon an analysis of the scree plot emd latent roots 
(eigen values) of the factors (Hair, Anderson, Tatham, 
&Black, 1992). This analysis involved analyzing the 
scree plot for "breaks" among the factors and select- 
ing only those suited. These experts were asked to 
analyze individual in dicators which "loaded on" (were 
correlated with) each of the five factors. Indicators 
which had factor loadings above .40 were listed for 
each factor (Ford, et al., 1986). The panel was asked 
to examine the list of indicators for each factor, and 
suggest a short title which represented the list of 
indicators. This information was then summarized 
by the researcher. 
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Table 5 

Spearman Rank Order Correlations of Respondent Groups Feasibility Rankings of 

Indicator Statements 



Respondent Group 


1. 


2. 


3. 


4. 5. 


1. Overall Group Rank 










2. Executive Directors SCOVEs 


.90 








3. State Legislators 


.87 


.75 






4. State Directors of Voc. Educ. 


.93 


.82 


.73 




5. Local Admin. 


.96 


.81 


.81 


.87 — 



8 



8 



The first extracted factor was named "em- 
ployer satisfaction" , and consisted of five indicator 
statements. The second extracted factor was named 
"economic and social benefits". Six indicators 
loaded on this factor. The third extracted factor was 
named "employment rates". Three indicators 
loaded on this factor. The fourth extracted factor 
was named "cognitive achievement". Three indi- 
cators loaded on this factor. The fifth extracted 
factor was named "individual and institutional 
educational effort". Six indicators loaded on this 
faCiOr. The named factors and their respective 
indicator lists are shown in Table 6. 

Research Question Six 

To answer this question, the semantic content 
of the five named factors was compared with compo- 
nents of the McCaslin (1990) evaluation framework. 
Table 7 graphically represents this comparison. The 
right-hand section of the firamework classifies the 
outcomes of vocational education — economic, educa- 
tional, and psychosocial. In order to determine the 
congruence of these factors with aspects of the Mc- 
Caslin framework, a direct semantic comparison of 
the factor (as named by the panel of experts) was 
made with specific subparts of the framework. The 
outcomes components of McCaslin's framework, along 
with all of the proposed measures of outcomes, are 
listed in the left-hand column. The factors identified 
in this study are Usted across the top row of Table 7. 
Individual indicators which comprise each factor are 
listed in the columns. Shaded areas in Table 7 
represent apparent congruence between a factor 
identified in this research and parts of McCaslin's 
framework. 

The first factor extracted from the importance 
ratings was named "employer satisfaction." This 
factor includes five indicators— three direct mea- 
sures of employer satisfaction with vocational edu- 
cation program graduates and two measures of gradu- 
ate ability and work ethic. These indicators are 
aUgned in Table 7 with those components of the 
McCaslin firamework which appear to have the most 
congruence. This factor seemed to match both eco- 
nomic and educational components of outcomes in 
the McCasUn firamework. The three direct measures 
of employer satisfaction clearly represent an eco- 
nomic outcome which McCaslin posited. The direct 
measure of student ability is an educational out- 
come. The indicator of work ethic is also a psychosocial 
outcome. When an indicator (such as work ethic) 
appears to have congruence with more than one 
component of the framework, it is shown twice in 
Table 7. 
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The second factor, "economic and social ben- 
efits", also had congruence with the outcomes sec- 
tion of the framework. Of the six indicators which 
comprised the factor, three were measures of eco- 
nomic outcomes and three were measures of the 
social outcomes of vocational education programs. 
Two of the economic indicators directly matched the 
economic part of the framework. Two of the 
psychosocial indicators directly matchedpsychosocial 
parts of the framework. As with the first factor, the 
congruence with McCaslin's framework was partial, 
not complete. 

The third and fourth factors each had high 
levels of congruence with the evaluative firamework. 
Factor three, "employment rates", directly matched 
some of the economic components of vocational edu- 
cation program outcomes. "Cognitive achievement," 
the fourth factor, directly matched the two of the 
educational portion of vocational education program 
outcomes. The fifth factor, "individual and institu- 
tional effort ' did not match any single component of 
the firamev.'ork well. Individual indicators firom this 
factor matched the economic, educational, and 
psychosocial components of the firamework. Indica- 
tors which loaded on this factor measure both pro- 
cesses and outcomes of vocational education. "Indi- 
vidual and institutional effort", as defined by the 
factor analysis, appeared to be less congruent with 
McCaslin's framework than the other four identified 
factors. Data collected in this research appeared to 
have some congruence with McCaslin's firamework 
for evaluating vocational education programs, based 
upon the importance ratings of the 38 indicator 
statements used in the survey. 

Discussion 

These findings echoed some of the recent rheto- 
ric involving American public education. In 1991, 
the President estabUshed a set of National Educa- 
tion Goals. One of these goals stated that 

"American students will leave . . . having 
demonstrated competency in challenging 
matter . . . and every school in America 
will ensure that all students learn to use 
their minds well, so they may beprepared 
for responsible citizenship, further learn- 
ing, and productive employment in our 
modern economy." (U.S. Department of 
Education, 1991, p. 3). 

Another of the goals stated: 
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Table 6 

Rotated Factor Matrix Loadings Order of Iwortance Ratings of Indicator Statements 
ON Oblique Factors (n=2Q7) 



Factx)r 1 — Employer Satisfaction Loading 

Employer satisfaction of program graduates work competency measured by opinion survey. .95 

Employer satisfaction of program graduates work habits measured by opinion survey. .92 

Employer satisfaction of program graduates training measured by opinion survey. .54 
Problem solving ability of program graduates measured by employer appraisal and teacher 

oDservations. .47 

Program graduates' work ethic as measured by survey of employers and teacher observation. .41 



Factor 2 — Economic and Social Benefits 

Student job success as measured by the average weekly wage of program graduates. .67 

The earnings of program graduates employed in jobs directly related to their training. .57 
The economic return on investment of program graduates as measured by the ratio of 

graduate earnings to program costs. 56 

Student citizenship as measured by the rate of graduate voter registration. .47 
The rate of program graduates who receive government assistance (ADC, General relief, 

or food stamps). -46 

Educational aspirations of program graduates measured by opinion survey. .43 



Factor 3 — Employment Rates 

Student participation in the labor force after completing the program. 
The employment rate of program graduates. 

The rate of student employment in jobs directly related to their training. 



.74 
.71 
.65 



Factor 4 — Cognitive Achievement 

Student achievement measured by a gain on a standardized test (pretest-posttest). .80 
Student achievement as measured by gain of students' basic skills on a standard proficiency 

test (pretest-posttest). -77 
Critical thinking ability of programi graduates as measured by a standardized critical 

thinking test. -44 



Factor 5 — Individual and Institutional Educational Effort 

A description of the counseling services available to the student. .65 

A description of the recruitment activities conducted by the program. .61 

Evidence of teacher performance measured by supervisor ratings. .46 

Student attendance measured by average days per school year missed by students. .44 
Program graduates' work ethic as measured by survey of employers and teacher observation. .42 

Educational aspirations of program graduates measured by opinion survey. .40 



"Everj' adult American will be literate 
and wiU possess the knowledge and skills 
necessary to compete in a global economy 
and exercise the rights and responsibili- 
ties of citizenship." 

This research identified measures of employer 
satisfaction, student achievement, the labor market 
outcomes of students, and student educational ad- 
vancement to be among the most important indica- 
tors for evaluating vocational education programs. 
Additionally, measures of student attendance, re- 
tention, achievement, learning, labor market out- 
comes, and employer satisfaction were rated as be- 
ing the most feasible indicators for evaluating 
secondary vocational education programs. AH of 
these measures fit within the two National Goals for 
American Education listed above. 



These findings also reflected recommendations 
which have been made for vocational education. 
Wirt, et. al. (1989) in the National Assessment of 
Vocational Education recommended that vocational 
education should: 

"Revise and rebuild the high school [sec- 
ondary] curriculum to upgrade skill levels 
and provide students with the mix of occu- 
pationaUy specific and transferable skills 
they need to get good jobs or to pursue 
further training and education at the 

postsecondary level [and] . . . Integrate 

high school academic and vocational cur- 
ricula so that students come to vocational 
programs well equipped with fundamental 
academic skills. Vocational courses should 
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also provide an applied context, based on 
broad and specific job training, that rein- 
forces eind enha nces academic skills and 
motivates students to excel in both aca- 
demic eind vocational courses. . . [and] . . . 
Expand efforts lo place students in good 
jobs that make full use of their vocational 
and academic training." (pp. iv-xv). 

The indicators identified as being both impor- 
tant and feasible were also consistent with public 
policy regarding vocational education. The Perkins 
Act specifically outlined the use of performance 
standards and measures for evaluatirg v x:ationaI 
education programs. These measures included mea- 
sures of academic achievement, educational advance- 
ment, and labor market outcomes of vocational edu- 
cation students. 

These findings supported past research in voca- 
tional education evaluations. Taylor, Darcy, and 
Bolland (1979) compiled a bibliography of evaluative 
£ ■ adies of vocational education programs. In par- 
ticular, studies which measured program graduate 
employment status, earnings, rates of further edu- 
cation, employer satisfaction, and student learning 
were among the most firequently used evaluative 
criteria Copa and Salem (1982) identified the follow- 
ing indicators as the most important for use in voca- 
tional education: employer satisfaction with the qual- 
ity of graduate work; tiienvmiber of graduates employed 
in occupations related to training; projected job open- 
ings in occupations related to training; number of 
graduates employed; occupations for which program 
is designed to provide training; performance of enroll- 
ees in program; and program cost. Respondents who 
participated in this research identuied similar mea- 
sures as being both important and feasible for use in 
evaluating vocational education programs. 

Burstern, Oakes, andGuiton (1992) maintained 
that educational effectiveness indicators must be 
developed which reflect both scientific and political 
realities. One such political reality is that informa- 
tion collected by educational indicators be relevant 
to all stakeholders of an educational system. In 
essence, some consensus among individuals at sev- 
eral levels of decision-mal mg — ^the legislature, ad- 
visory groups, state officials, and local administra- 
tors — must be achieved in order for an educational 
indicator system to be viable. One criticism of 
vocational educators has been that they have not 
been proactive in determining some of the standards 
against which vocational education programs should 
be judged. This research showed high levels of 
agreement between the respondent groups in terms 
of their importance ranking of vocational education 



evaluation indicator statements. There was also a 
high level of agreement between the respondent 
groups in terms of their feasibility rankings of the 
vocational education evaluation indicator statements. 
These data showed that among these groups of 
respondents, there was some level of consensu- 
between different groups of stakeholders of voca- 
tional education in terms of what types of indicators 
are important and feasible for use in evaluating 
vocational education programs. 

Ck»pa and Scholl,(1983) stated that indicators 
for evaluating vocational education must be tied to a 
clear, conceptual framework of the vocational school- 
ing process. Bryk and Hermanson (1993) wrote of 
the necessity of a clear, conceptual coherence among 
the different components of an indicator system. 
Thty further mziintained that scientific social re- 
search must confirm or disconfirm these relation- 
ships. To date, little research has been done on a 
national scale which has tried to empirically exEun- 
ine factors which comprise vocational education 
quality indicators. DeNeufville (1975) held that at 
some point in the "lifespan" of an indicator system, 
the scientific viability of the conceptual framework 
which that indicator system purports to represent 
must be exEunined. Van Dalen (1979) wrote that 
exploratory research was a necessary first step in 
the acceptance or disconfrrrjiation of a theory. Hair, 
et al. (1992) noted that exploratory factor analysis is 
often necessary when attempting to establish con- 
ceptual relationships among a large set of variables. 
This exploratory study identified five factors ac- 
counted for 38.5% of the common variance in the 
importance ratings of the indicators statements. 
Four of these factors were deemed "congruent" with 
the outcomes component of the McCaslin fi:ame- 
work. Two of the factors had almost perfect congru- 
ence (see Figure 7). When the findings of this study 
(described above) are contexted with some of the 
literature regarding educational reform, social and 
educational indicators, vocational education evalua- 
tion, and exploratory research, thefollowing conclu- 
sions are drawn: 

Conclusion One. Indicators which measure 
employer satisfaction, student achievement, the la- 
bor market outcomes of students, and the educa- 
tional advancement of students are the most impor- 
tant measures for use in vocational education 
evaluation. 

Conclusion Two. Measures of student atten- 
dance, retention, achievement, learning, labor mar- 
ket outcomes, and employer satisfaction are rated as 
the most feasible indicators for evaluating secondary 
vocational education programs. 
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Conclusion Three. Sixteen indicators had 
both relatively high importance and feasibility rat- 
ings, giving these indicators the greatest potential 
for immediate use in evaluating vocational educa- 
tion programs. These indicators included measures 
of labor outcomes, employer satisfaction, student 
educational advancement, studentachievement, stu- 
dent learning, retention, satisfaction, work ethic, 
and a measure of school planning. Of these 16 
indicators, 15 were measures of outcomes of voca- 
tional education, and one a measure of a vocational 
education process. 

Conclu"' ^ Four. There was ahigh degree of 
consensus among the respondent groups (executive 
directors of state councils of education , state legisla- 
tors, state directors of vocational education, and 
local administrators of vocational education) in terms 
of their importance rating of vocational education 
quality indicators. There was also a degree of con- 
sensus in terms of their feasibility ratings of these 
indicators. Respondents to this survey agreed about 
what types o^ quality indicators were both impor- 
t£int t nd feasible for use in evaluating vocational 
education programs. 

Conclusion Five. The importance ratings of 
vocational education quedity indicators contained 
underlying conceptual fact ars. These factors were: 
"employer satisfaction", "economic and social ben- 
efits" , "employment rates" , "cognitive achievement" , 
and "individual and institutional educational ef- 
fort". These factors represent constructs for evalu- 
ating vocational education programs and provide a 
first step in empirically identif)TLng conceptual rela- 
tionships for evaluating vocational education pro- 
grzims. 

Conclusion Six. Four of the identified factors 
had congruence with McCaslin's framework for 
evaluating vocational education programs. Specifi- 
cally, these factors had congruence with the "out- 
comes" component of the model. Data collected in 
this research empirically gave support to McCaslin's 
evaluation framework. The outcomes of vocational 
education programs were conceptually comprised of 
economic, educational, and psychosocial components. 
Evidence from this study suggests that these com- 
ponents have scientific merit. 

Recommendations for Practice 

The conclusions of this research provided the 
basis for several recommendations which are appli- 
cable to the practice of vocational education evalu- 
ation. These recommendations include: 
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1. The use of indicators for evaluating vocational 
education program outcomes should bepromoted 
by those who enact vocational education policy. 
In particular, state imdfederallegislatures should 
encourage the use of measures of labor meirket 
outcomes of students, employer satisfaction with 
vocational education students, student educa- 
tional advancement, student achievement, stu- 
dent learning, student retention, student satis- 
faction, and student work ethic. 

2. Those who administer and advise vocational 
education programs, particulctrly state and local 
directors, and members of advisory councils 
should promote using the 'Outcome indicators 
mentioned above. 

3. Since high levels of agreement were appeirent in 
both these data and in the literature, individuals 
at all levels of vocational education policy should 
capitalize upon the common ground which exists 
as to the efficacy of these outcome indicators. 
When competing for scarce public resources, 
those concerned with vocational education can 
take advantage of the fact that there was agree- 
ment as to which outcomes are important. This 
agreement is often lacking in educational re- 
form, and this lack of agreement subsequently 
hinders reform efforts. 

4. Those who administer, fund, and evaluate voca- 
tional education program should consider using 
the constructs of employer satisfaction, employ- 
ment rates, cognitive achievement, and economic 
and social benefits when conceptualizing and 
contextualizing evaluative information regrad- 
ing vocational education programs. 

5. Those who administer, fund, and evaluate voca- 
tional education programs should also consider 
the use of the outcomes component of McCaslin's 
evaluative framework in interpreting the evalu- 
ation results of vocational education progrzims. 

Reccmmendations for Theory 

The conclusions of this research provided the 
basis for several recommendations which were ap- 
plicable to the study of the theories of vocational 
education. These recommendations included: 

1. Evaluators of vocational education programs 
should consider the use of the outcome con- 
structs employer satisfaction, employment rates, 
cognitive achievement, and economic and social 
benefits when designing evaluations of voca- 
tional education programs. These constructs 
have a degree of empirical validity. Use of vaUd 
constructs when operationalizing measurements 
in evaluations is critical. 
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2. Evaluators of vocational education programs 
should also consider the use of the 16 indicators 
described in Conclusion Four when designing 
evaluations of vocational education. These indica- 
tors were identified as having perhaps the greatest 
potential for use. 

3. Scholars of vocational education should consider 
the outcome factors identified in this research as 
a contribution to the knowledge base regarding 
vocational education. Since relatively few at- 
tempts at empirical theory building exist in 
vocational education, this research may provide 
scholars with a philosophical point of departure 
for scholarly debate and inquiry within the dis- 
cipline of vocational education. 

4, Scholars of vocational education should likewise 
examine the sixteen indicators identified in con- 
clusion four as another contribution to the knowl- 
edge base of vocational education. Debate on the 
efficacy of these indicators could open important 
areas of philosophical inquiry with vocational 
education. 

5, Finally, the discipline of vocational education 
should give serious consideration to McCasliii's 
evaluation framework for vocational education. 
These data show that the outcomes portion of the 
framework have a degree of empirical congru- 
ence. These findings should lead scholars and 
practitioners to further examine the conceptual 
coherence of this framework. 

Recommendations for Research 

The conclusions of this research provided the 
basis for several recommendations which are in- 
tended to stimulate further scientific research in 
vocational education.These recommendations included: 

1. Research should replicate this study on other 
groups of vocational education stakeholders, such 
as teachers, students, parents, and employers. 
These groups represent important sources of 
data which may confibnm or disconfirm this siudy* s 
findings. Federed legislators must be included in 
follow-up research on this topic. Alternative 
methods fc gathering data must be considered 
when Federal legislators are used. When pos- 
sible, probabilistic sampUng procedures should 
be employed to increase the generalizability of 
these studies. 

2. Research should actually examine the use and 
implementation of the indicators identified in 
this study. Case study research may be able to 
explore in greater the efficacy of the use of 
systems of indicators in vocational education 
evaluation. 



3. Studies designed to estimate other aspects of the 
McCasIin fr^ework should be conducted. This 
research only identified factors related to the 
outcomes portion of the model. The coherence of 
the needs and processes portions of the model 
need to be examined. Also, the proposed interac- 
tions between the different parts of the model 
should be investigated. 

4. Feasibility studies which estimate the technical 
requirements gnd limitations of indicators iden- 
tified in this study, as well as some of the re- 
sources required to implement the use of these 
indicators should be conducted. 

5. Other research should examine aspects of the 
level of agreement in terms of indicator impor- 
tance and feasibility in this research. Group 
techniques, or Delphi methodologies could re- 
fine and extend the areas of consensus between 
different groups of policy makers which was 
found in this study. 

6. Finally, research should the examine the types 
of indicators which have efficacy for evaluating 
postsecondary vocational education programs. 
This research examined indicators only within 
the context of secondary education. There are 
distinct differences between secondary and 
postsecondary vocational education. Logically, 
the indicators effective for evaluating postsec- 
ondary vocational education may be different 
than those for secondary vocational education. 



References 

Asche. M. (1990). Standards and measures of perfor- 
mance indicators of quality for Virginia voca- 
tional education programs. Blacksburg: Vir- 
ginia Polytechnic Institute and State University, 
Division of Vocational and Technical Education. 

Bryk, A. S., & Hermanson, K. L., (1993). Educa- 
tional indicator systems: Observations on their 
structure, interpretation, anduse. in L. Darling- 
Hammond (Ed.), Review of Research in Educa- 
tion, pp. 451-484. Washington, D.C.: American 
Educational Research Association. 

Burstein. L.; Oakes, J.; & Guiton, G. (1992). Educa- 
tion indicators, in M. Alkrn, (Ed.), Encyclopedia 
of educational research, sixth edition, pp. 409- 
415. New Yofk: MacMillan Publishing Co. 

Carl D. Perkins Vocational Education Act. (Public 
Law 98-524) (1984). 98th Congress: An Act to 
Amend the Vocational Education Act of 1963. 
Washington, DC: U.S. Government Printing 
Office. 

Carl D. Perkins Vocational and AppUed Technology 
Education Act. Amendments of 1990 (PubUc 



ERIC 



15 



14 



Law 100-392) Section 115. (1990). Washington. 
DC: U.S. Government Printing Office. 

Copa. G. H., & Salem, M. N. (1982) Vital statistics 
for planning, review, and public information 
about vocational education. Minneapolis: Min- 
nesota Research and Development Center for 
VocationalEducation.Departmentof Vocational 
and Technical Education, The University of 
Minnesota (ERIC Document Reproduction Ser- 
vice Number ED 224 995). 

Copa, G. H., & Scholl, S. (1983) Developing voca- 
tional education indicators: Some steps in mov- 
ing from selection to use. St. Paul: Minnesota 
Research and Development Center for Voca- 
tional Education. (ERIC Document Reproduc- 
tion Service No. ED 235 385). 

Darcy , R. L. , ( 1 979) . Vocational education outcomes: 
Perspectives for evaluation. Columbus: The 
National Center for Research in Vocational Edu- 
cation. (ERIC Document Reproduction Service 
No. ED 177 323). 

DeNeufville, J. (1975). Social indicators and public 
policy: Interactive processes of design and appli- 
cation. Amsterdam: Elsevier Scientific Publish- 
ing Co. 

Hair, J.F.; Anderson, R.E.; Tatham R.L.; & Black, 
W.C., (1992). Multivariate data analysis with 
readings. New York: Macmillan. 

Hoachlander, G., & Rahn, M. L., (1992). Perfor- 
mance measures and standards for vocational 
education: 1991 survey results. Berkeley: Na- 
tional Center for Resejirch in Vocational Educa- 
tion. 

Johnston, W. B. (1987). Workforce 2000: Work and 
workers for the 21st century. Indianapolis: 
Hudson Institute. 

McCaslin, N. L., (1990). A framework for evaluating 
local vocational education programs. Colum- 
bus: The Ohio State University. (ERIC Docu- 



ment Reproduction Service No. ED 327 738). 

McCaslin. N. L., & Headley, S., (1993). A national 
study of approved state systems of performance 
measures and standards for vocational educa- 
tion. Columbus: The Ohio State University, 
Comprehensive Vocational Education Program. 

National Commission on Excellence in Education. 
(1983). A nation at risk. Washington, DC: U.S. 
Department of Education. 

Raizen, S.A., & Jones. L.V. (1985). Indicator of 
precollege education in science and mathemat- 
ics: A preliminary review. Washington, DC: 
National Academy Press. 

Taylor. CM.: Darcy, R.L.; & BoUand, K.A. (1979). 
Vocational education outcomes: Annotated bib- 
liography of related literature. Columbus: Na- 
tional Center for Research in Vocational Educa- 
tion. (ERIC Document Reproduction Service 
No. ED 177 372). 

U.S. Department of Education. (1991). Education 
counts: An indicator system to monitor the 
nations educational health. Washington, DC: 
Author. 

U.S. Department of Education. (1991). America 
2000: An education strategy. Washington, DC: 
Author. 

White, B. (1990) Vocational education effectiveness 
indicators. Honolulu: University of Hawaii 
Office of State Director for Vocational Educa- 
tion. (ERIC Document Reproduction Service 
No. ED 324 408). 

Wirt, J. G.; Muraskin. L. D.; Goodwin, D.A; & 
Meyer. R.H. (1989). National assessment of 
vocational education: Summary of findings and 
recommendations. Washington. DC: U.S. De- 
partment of Education. 

VanDalen, D.B. (1970). Understanding educational 
research. New York:' McGraw-Hill. 



SUMMARY OF RESEARCH SERIES 



Recent national reform reports have carried a theme of holding schools accountable 
for student achievement, attainment, and retention of requisite knowledge, sHUs, and 
attitudes. If educational decision-makers are going to use vocational education indica- 
tors for program improvement, then a systematic effort must be undertaken to determine 
which indicators are most relevant and useful. This research validates a hst of indicator 
statements for use in evaluating secondary vocational education programs. It shoiild be 
of interest to vocational education teachers, local vocational administrators, federal 
legislators, state legislators, state directors of vocational education, and executive 
directors of state councils of vocational education. 
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the purpose of this series to make useful knowledge from such research available to 
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